20 research outputs found
Attributes2Classname: A discriminative model for attribute-based unsupervised zero-shot learning
We propose a novel approach for unsupervised zero-shot learning (ZSL) of
classes based on their names. Most existing unsupervised ZSL methods aim to
learn a model for directly comparing image features and class names. However,
this proves to be a difficult task due to dominance of non-visual semantics in
underlying vector-space embeddings of class names. To address this issue, we
discriminatively learn a word representation such that the similarities between
class and combination of attribute names fall in line with the visual
similarity. Contrary to the traditional zero-shot learning approaches that are
built upon attribute presence, our approach bypasses the laborious
attribute-class relation annotations for unseen classes. In addition, our
proposed approach renders text-only training possible, hence, the training can
be augmented without the need to collect additional image data. The
experimental results show that our method yields state-of-the-art results for
unsupervised ZSL in three benchmark datasets.Comment: To appear at IEEE Int. Conference on Computer Vision (ICCV) 201
Zero-Shot Object Detection by Hybrid Region Embedding
Object detection is considered as one of the most challenging problems in
computer vision, since it requires correct prediction of both classes and
locations of objects in images. In this study, we define a more difficult
scenario, namely zero-shot object detection (ZSD) where no visual training data
is available for some of the target object classes. We present a novel approach
to tackle this ZSD problem, where a convex combination of embeddings are used
in conjunction with a detection framework. For evaluation of ZSD methods, we
propose a simple dataset constructed from Fashion-MNIST images and also a
custom zero-shot split for the Pascal VOC detection challenge. The experimental
results suggest that our method yields promising results for ZSD
Fine-Grained Object Recognition and Zero-Shot Learning in Remote Sensing Imagery
Fine-grained object recognition that aims to identify the type of an object
among a large number of subcategories is an emerging application with the
increasing resolution that exposes new details in image data. Traditional fully
supervised algorithms fail to handle this problem where there is low
between-class variance and high within-class variance for the classes of
interest with small sample sizes. We study an even more extreme scenario named
zero-shot learning (ZSL) in which no training example exists for some of the
classes. ZSL aims to build a recognition model for new unseen categories by
relating them to seen classes that were previously learned. We establish this
relation by learning a compatibility function between image features extracted
via a convolutional neural network and auxiliary information that describes the
semantics of the classes of interest by using training samples from the seen
classes. Then, we show how knowledge transfer can be performed for the unseen
classes by maximizing this function during inference. We introduce a new data
set that contains 40 different types of street trees in 1-ft spatial resolution
aerial data, and evaluate the performance of this model with manually annotated
attributes, a natural language model, and a scientific taxonomy as auxiliary
information. The experiments show that the proposed model achieves 14.3%
recognition accuracy for the classes with no training examples, which is
significantly better than a random guess accuracy of 6.3% for 16 test classes,
and three other ZSL algorithms.Comment: G. Sumbul, R. G. Cinbis, S. Aksoy, "Fine-Grained Object Recognition
and Zero-Shot Learning in Remote Sensing Imagery", IEEE Transactions on
Geoscience and Remote Sensing (TGRS), in press, 201
Weakly Supervised Object Localization with Multi-fold Multiple Instance Learning
Object category localization is a challenging problem in computer vision.
Standard supervised training requires bounding box annotations of object
instances. This time-consuming annotation process is sidestepped in weakly
supervised learning. In this case, the supervised information is restricted to
binary labels that indicate the absence/presence of object instances in the
image, without their locations. We follow a multiple-instance learning approach
that iteratively trains the detector and infers the object locations in the
positive training images. Our main contribution is a multi-fold multiple
instance learning procedure, which prevents training from prematurely locking
onto erroneous object locations. This procedure is particularly important when
using high-dimensional representations, such as Fisher vectors and
convolutional neural network features. We also propose a window refinement
method, which improves the localization accuracy by incorporating an objectness
prior. We present a detailed experimental evaluation using the PASCAL VOC 2007
dataset, which verifies the effectiveness of our approach.Comment: To appear in IEEE Transactions on Pattern Analysis and Machine
Intelligence (TPAMI
Image Captioning with Unseen Objects
Image caption generation is a long standing and challenging problem at the
intersection of computer vision and natural language processing. A number of
recently proposed approaches utilize a fully supervised object recognition
model within the captioning approach. Such models, however, tend to generate
sentences which only consist of objects predicted by the recognition models,
excluding instances of the classes without labelled training examples. In this
paper, we propose a new challenging scenario that targets the image captioning
problem in a fully zero-shot learning setting, where the goal is to be able to
generate captions of test images containing objects that are not seen during
training. The proposed approach jointly uses a novel zero-shot object detection
model and a template-based sentence generator. Our experiments show promising
results on the COCO dataset.Comment: To appear in British Machine Vision Conference (BMVC) 201
Zero-Shot Sign Language Recognition: Can Textual Data Uncover Sign Languages?
We introduce the problem of zero-shot sign language recognition (ZSSLR),
where the goal is to leverage models learned over the seen sign class examples
to recognize the instances of unseen signs. To this end, we propose to utilize
the readily available descriptions in sign language dictionaries as an
intermediate-level semantic representation for knowledge transfer. We introduce
a new benchmark dataset called ASL-Text that consists of 250 sign language
classes and their accompanying textual descriptions. Compared to the ZSL
datasets in other domains (such as object recognition), our dataset consists of
limited number of training examples for a large number of classes, which
imposes a significant challenge. We propose a framework that operates over the
body and hand regions by means of 3D-CNNs, and models longer temporal
relationships via bidirectional LSTMs. By leveraging the descriptive text
embeddings along with these spatio-temporal representations within a zero-shot
learning framework, we show that textual data can indeed be useful in
uncovering sign languages. We anticipate that the introduced approach and the
accompanying dataset will provide a basis for further exploration of this new
zero-shot learning problem.Comment: To appear in British Machine Vision Conference (BMVC) 201
HybridAugment++: Unified Frequency Spectra Perturbations for Model Robustness
Convolutional Neural Networks (CNN) are known to exhibit poor generalization
performance under distribution shifts. Their generalization have been studied
extensively, and one line of work approaches the problem from a
frequency-centric perspective. These studies highlight the fact that humans and
CNNs might focus on different frequency components of an image. First, inspired
by these observations, we propose a simple yet effective data augmentation
method HybridAugment that reduces the reliance of CNNs on high-frequency
components, and thus improves their robustness while keeping their clean
accuracy high. Second, we propose HybridAugment++, which is a hierarchical
augmentation method that attempts to unify various frequency-spectrum
augmentations. HybridAugment++ builds on HybridAugment, and also reduces the
reliance of CNNs on the amplitude component of images, and promotes phase
information instead. This unification results in competitive to or better than
state-of-the-art results on clean accuracy (CIFAR-10/100 and ImageNet),
corruption benchmarks (ImageNet-C, CIFAR-10-C and CIFAR-100-C), adversarial
robustness on CIFAR-10 and out-of-distribution detection on various datasets.
HybridAugment and HybridAugment++ are implemented in a few lines of code, does
not require extra data, ensemble models or additional networks.Comment: Accepted to ICCV 202
Segmentation Driven Object Detection with Fisher Vectors
International audienceWe present an object detection system based on the Fisher vector (FV) image representation computed over SIFT and color descriptors. For computational and storage efficiency, we use a recent segmentation-based method to generate class-independent object detection hypotheses, in combination with data compression techniques. Our main contribution is a method to produce tentative object segmentation masks to suppress background clutter in the features. Re-weighting the local image features based on these masks is shown to improve object detection significantly. We also exploit contextual features in the form of a full-image FV descriptor, and an inter-category rescoring mechanism. Our experiments on the VOC 2007 and 2010 datasets show that our detector improves over the current state-of-the-art detection results
Multi-fold MIL Training for Weakly Supervised Object Localization
International audienceObject category localization is a challenging problem in computer vision. Standard supervised training requires bounding box annotations of object instances. This time-consuming annotation process is sidestepped in weakly supervised learning. In this case, the supervised information is restricted to binary labels that indicate the absence/presence of object instances in the image, without their locations. We follow a multiple-instance learning approach that iteratively trains the detector and infers the object locations in the positive training images. Our main contribution is a multi-fold multiple instance learning procedure, which prevents training from prematurely locking onto erroneous object locations. This procedure is particularly important when high-dimensional representations, such as the Fisher vectors, are used. We present a detailed experimental evaluation using the PASCAL VOC 2007 dataset. Compared to state-of-the-art weakly supervised detectors, our approach better localizes objects in the training images, which translates into improved detection performance